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                        Tools for DNS debugging

Status of this Memo

   This memo provides information for the Internet community.  This memo
   does not specify an Internet standard of any kind.  Distribution of
   this memo is unlimited.

Abstract

   Although widely used (and most of the times unnoticed), DNS (Domain
   Name System) is too much overlooked, in the sense that people,
   especially administrators, tend to ignore possible anomalies as long
   as applications that need name-to-address mapping continue to work.
   This document presents some tools available for domain administrators
   to detect and correct those anomalies.

1. Introduction

   Today more than 3,800,000 computers are inter-connected in a global
   Internet [1], comprising several millions of end-users, able to reach
   any of those machines just by naming it.  This facility is possible
   thanks to the world widest distributed database, the Domain Name
   System, used to provide distributed applications various services,
   the most notable one being translating names into IP addresses and
   vice-versa.  This happens when you do an FTP or Telnet, when your
   gopher client follows a link to some remote server, when you click on
   a hypertext item and have to reach a server as defined by the URL,
   when you talk to someuser@some.host, when your mail has to be routed
   through a set to gateways before it reaches the final recipient, when
   you post an article to Usenet and want it propagated all over the
   world.  While these may be the most visible uses of DNS, a lot more
   applications rely on this system to operate, e.g., network security,
   monitoring and accounting tools, just to mention a few.

   DNS owes much of its success to its distributed administration.  Each
   component (called a zone, the same as a domain in most cases), is
   seen as an independent entity, being responsible for what happens
   inside its domain of authority, how and what information changes and
   for letting the tree grow downwards, creating new components.
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   On the other hand, many inconsistencies arise from this distributed
   nature: many administrators make mistakes in the way they configure
   their domains and when they delegate authority to sub-domains; many
   of them don't even know how to do these things properly, letting
   problems last and propagate.  Also, many problems occur due to bad
   implementations of both DNS clients and servers, especially very old
   ones, either by not following the standards or by being error prone,
   creating or allowing many of the above problems to happen.

   All these anomalies make DNS less efficient than it could be, causing
   trouble to network operations, thus affecting the overall Internet.
   This document tries to show how important it is to have DNS properly
   managed, including what is already in place to help administrators
   taking better care of their domains.

2. DNS debugging

   To help finding problems in DNS configurations and/or implementations
   there is a set of tools developed specifically for this purpose.
   There is probably a lot of people in charge of domain administration
   having no idea of these tools (and, worse, not aware of the anomalies
   that may exist in their configurations).  What follows is a
   description of some of these programs, their scope, motivations and
   availability, and is hoped to serve as an introduction to the subject
   of DNS debugging, as well as a guide to those who are looking for
   something to help them finding out how healthy their domains and
   servers are.

   Some prior knowledge from the reader is assumed, both on DNS basics
   and some other tools (e.g., dig and nslookup), which are not analyzed
   in detail here; hopefully they are well-known enough from daily
   usage.

2.1. Host

   Host is a program used to retrieve DNS information from name servers.
   This information may be used simply to get simple things like
   address-to-name mapping, or some more advanced purposes, e.g.,
   performing sanity checks on the data.  It was created at Rutgers
   University, but then Eric Wassenaar from Nikhef did a major rewrite
   and still seems to be actively working on improving it.  The program
   is available from ftp://ftp.nikhef.nl/pub/network/host_YYMMDD.tar.Z
   (YYMMDD is the date of the latest release).

   By default, host just maps host names to Internet addresses, querying
   the default servers or some specific one.  It is possible, though, to
   get any kind of data (resource records) by specifying different query
   types and classes and asking for verbose or debugging output, from
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   any name server.  You can also control several parameters like
   recursion, retry times, timeouts, use of virtual circuits vs.
   datagrams, etc., when talking to name servers.  This way you can
   simulate a resolver behavior, in order to find any problems
   associated with resolver operations (which is to say, any application
   using the resolver library).  As a query program it may be as
   powerful as others like nslookup or dig.

   As a debugger, host analyzes some set of the DNS space (e.g., an
   entire zone) and produces reports with the results of its operation.
   To do this, host first performs a zone transfer, which may be
   recursive, getting information from a zone and all its sub-zones.
   This data is then analyzed as requested by the arguments given on the
   command line.  Note that zone transfers are done by contacting
   authoritative name servers for that zone, so it must be possible to
   make this kind of request from such servers: some of them refuse zone
   transfers (except from secondaries) to avoid congestion.

   With host you may look for anomalies like those concerning authority
   (e.g., lame delegations, described below) or some more exotic cases
   like extrazone hosts (a host of the form host.some.dom.ain, where
   some.dom.ain is not a delegated zone of dom.ain).  These errors are
   produced upon explicit request on the command line, but you may get a
   variety of other error messages as a result of host's operations,
   something like secondary effects.  These may be mere warnings (which
   may be suppressed) or serious errors - in fact, warning messages are
   not that simple, most of them are due to misconfigured zones, so it
   might not be a good idea to just ignore them.

   Error messages have to do with serious anomalies, either with the
   packets exchanged with the queried servers (size errors, invalid
   ancounts, nscounts and the like), or others related to the DNS
   information itself (also called "status messages" in the program's
   documentation): inconsistencies between SOA records as shown by
   different servers for a domain, unexpected address-to-name mappings,
   name servers not responding, not reachable, not running or not
   existing at all, and so on.

   Host performs all its querying on-line, i.e., it only works with data
   received from name servers, which means you have to query a name
   server more than once if you want to get different kinds of reports
   on some particular piece of data.  You can always arrange arguments
   in such a way that you get all information you want by running it
   once, but if you forget something or for any reason have to run it
   again, this means extra zone transfers, extra load on name servers,
   extra DNS traffic.
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   Host is an excellent tool, if used carefully.  Like most other
   querying programs it may generate lots of traffic, just by issuing a
   simple command.  Apart from that, its resolver simulation and debug
   capabilities make it useful to find many common and some not so
   common DNS configuration errors, as well as generate useful reports
   and statistics about the DNS tree.  As an example, RIPE (Reseaux IP
   Europeens) NCC uses it to generate a monthly european hostcount,
   giving an overview of the Internet usage evolution in Europe.  Along
   with these counts, error reports are generated, one per country, and
   the whole information is made available in the RIPE archive.

2.2. Dnswalk

   Dnswalk is a DNS debugger written in Perl by David Barr, from
   Pennsylvania State University.  You'll find the latest version at
   ftp://ftp.pop.psu.edu/pub/src/dnswalk.  With the software comes a
   small document where the author points some useful advice so it may
   be worth reading it.

   The program checks domain configurations stored locally, with data
   arranged hierarchically in directories, resembling the DNS tree
   organization of domains.  To set up this information dnswalk may
   first perform zone transfers from authoritative name servers. You can
   have a recursive transfer of a domain and its sub-domains, though you
   should be careful when doing this, as it may generate a great amount
   of traffic.  If the data is already present, dnswalk may skip these
   transfers, provided that it is up to date.

   Dnswalk looks for inconsistencies in resource records, such as MX and
   aliases pointing to aliases or to unknown hosts, incoherent PTR, A
   and CNAME records, invalid characters in names, missing trailing
   dots, unnecessary glue information, and so on.  It also does some
   checking on authority information, namely lame delegations and
   domains with only one name server.  It is easy to use, you only have
   to specify the domain to analyze and some optional parameters and the
   program does the rest.  Only one domain (and its sub-domains, if
   that's the case) can be checked at a time, though.

   While in the process of checking data, dnswalk uses dig and resolver
   routines (gethostbyXXXX from the Perl library) a lot, to get such
   data as authority information from the servers of the analyzed
   domains, names from IP addresses so as to verify the existence of PTR
   records, aliases and so on.  So, besides the zone transfers you may
   count on some more extra traffic (maybe not negligible if you are
   debugging a relatively large amount of data and care about query
   retries and timeouts), just by running the program.
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2.3. Lamers

   A lame delegation is a serious error in DNS configurations, yet a
   (too) common one.  It happens when a name server is listed in the NS
   records for some domain and in fact it is not a server for that
   domain.  Queries are thus sent to the wrong servers, who don't know
   nothing (at least not as expected) about the queried domain.
   Furthermore, sometimes these hosts (if they exist!) don't even run
   name servers.  As a result, queries are timed out and resent, only to
   fail, thus creating (more) unnecessary traffic.

   It's easy to create a lame delegation: the most common case happens
   when an administrator changes the NS list for his domain, dropping
   one or more servers from that list, without informing his parent
   domain administration, who delegated him authority over the domain.
   From now on the parent name server announces one or more servers for
   the domain, which will receive queries for something they don't know
   about.  (On the other hand, servers may be added to the list without
   the parent's servers knowing, thus hiding valuable information from
   them - this is not a lame delegation, but shouldn't happen either.)
   Other examples are the inclusion of a name in an NS list without
   telling the administrator of that host, or when a server suddenly
   stops providing name service for a domain.

   To detect and warn DNS administrators all over the world about this
   kind of problem, Bryan Beecher from University of Michigan wrote
   lamers, a program to analyze named (the well-known BIND name server)
   logging information [2].  To produce useful logs, named was applied a
   patch to detect and log lame delegations (this patch was originally
   written by Don Lewis from Silicon Systems and is now part of the
   latest release of BIND thanks to Bryan Beecher, so it is expected to
   be widely available in the near future).  Lamers is a small shell
   script that simply scans these logs and reports the lame delegations
   found.  This reporting is done by sending mail to the hostmasters of
   the affected domains, as stated in the SOA record for each of them.
   If this is not possible, the message is sent to the affected name
   servers' postmasters instead.  Manual processing is needed in case of
   bounces, caused by careless setup of those records or invalid
   postmaster addresses.  A report of the errors found by the U-M
   servers is also posted twice a month on the USENET newsgroup
   comp.protocols.tcp-ip.domains.

   If you ever receive such a report, you should study it carefully in
   order to find and correct problems in your domain, or see if your
   servers are being affected by the spreading of erroneous information.
   Better yet, lamers could be run on your servers to detect more lame
   delegations (U-M can't see them all!).  Also, if you receive mail
   reporting a lame delegation affecting your domain or some of your
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   hosts, please don't just ignore it or flame the senders.  They're
   really trying to help!

   You can get lamers from ftp://terminator.cc.umich.edu/dns/lame-
   delegations.

2.4. DOC

   Authority information is one of the most significant parts of the DNS
   data, as the whole mechanism depends on it to correctly traverse the
   domain tree.  Incorrect authority information leads to problems such
   as lame delegations or even, in extreme cases, the inaccessibility of
   a domain.  Take the case where the information given about all its
   name servers is incorrect: being unable to contact the real servers
   you may end up being unable to reach anything inside that domain.
   This may be exaggerated, but if you're on the DNS business long
   enough you've probably have seen some enlightened examples of this
   scenario.

   To look for this kind of problems Paul Mockapetris and Steve Hotz,
   from the Information Sciences Institute, wrote a C-shell script
   called DOC (Domain Obscenity Control), an automated domain testing
   tool that uses dig to query the appropriate name servers about
   authority for a domain and analyzes the responses.

   DOC limits its analysis to authority data since the authors
   anticipated that people would complain about such things as invasion
   of privacy.  Also, at the time it was written most domains were so
   messy that they thought there wouldn't be much point in checking
   anything deeper until the basic problems weren't fixed.

   Only one domain is analyzed each time: the program checks if all the
   servers for the parent domain agree about the delegation information
   for the domain.  DOC then picks a list of name servers for the domain
   (obtained from one of the parent's servers) and starts checking on
   their information, querying each of them: looks for the SOA record,
   checks if the response is authoritative, compares the various records
   retrieved, gets each one's list of NS, compares the lists (both among
   these servers and the parent's), and for those servers inside the
   domain the program looks for PTR records for them.

   Due to several factors, DOC seems to have frozen since its first
   public release, back in 1990.  Within the distribution there is an
   RFC draft about automated domain testing, which was never published.
   Nevertheless, it may provide useful reading.  The software can be
   fetched from ftp://ftp.uu.net/networking/ip/dns/doc.2.0.tar.Z.
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2.5. DDT

   DDT (Domain Debug Tools) is a package of programs to scan DNS
   information for error detection, developed originally by Jorge Frazao
   from PUUG - Portuguese UNIX Users Group and later rewritten by the
   author, at the time at the Faculty of Sciences of University of
   Lisbon.  Each program is specialized in a given set of anomalies: you
   have a checker for authority information, another for glue data, mail
   exchangers, reverse-mappings and miscellaneous errors found in all
   kinds of resource records.  As a whole, they do a rather extensive
   checking on DNS configurations.

   These tools work on cached DNS data, i.e., data stored locally after
   performing zone transfers (presently done by a slightly modified
   version of BIND's named-xfer, called ddt-xfer, which allows recursive
   transfers) from the appropriate servers, rather than querying name
   servers on-line each time they run.  This option was taken for
   several reasons [3]: (1) efficiency, since it reads data from disk,
   avoiding network transit delays, (2) reduced network traffic, data
   has to be fetched only once and then run the programs over it as many
   times as you wish and (3) accessibility - in countries with limited
   Internet access, as was the case in Portugal by the time DDT was in
   its first stages, this may be the only practical way to use the
   tools.

   Point (2) above deserves some special considerations: first, it is
   not entirely true that there aren't additional queries while
   processing the information, one of the tools, the authority checker,
   queries (via dig) each domain's purported name servers in order to
   test the consistency of the authority information they provide about
   the domain.  Second, it may be argued that when the actual tests are
   done the information used may be out of date.  While this is true,
   you should note that this is the DNS nature, if you obtain some piece
   of information you can't be sure that one second later it is still
   valid.  Furthermore, if your source was not the primary for the
   domain then you can't even be sure of the validity in the exact
   moment you got it in the first place.  But experience shows that if
   you see an error, it is likely to be there in the next version of the
   domain information (and if it isn't, nothing was lost by having
   detected it in the past).  On the other side, of course there's
   little point in checking one month old data...

   The list of errors looked for includes lame delegations, version
   number mismatches between servers (this may be a transient problem),
   non-existing servers, domains with only one server, unnecessary glue
   information, MX records pointing to hosts not in the analyzed domain
   (may not be an error, it's just to point possibly strange or
   expensive mail-routing policies), MX records pointing to aliases, A
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   records without the respective PTR and vice-versa, missing trailing
   dots, hostnames with no data (A or CNAME records), aliases pointing
   to aliases, and some more.  Given the specialized nature of each
   tool, it is possible to look for a well defined set of errors,
   instead of having the data analyzed in all possible ways.

   Except for ddt-xfer, all the programs are written in Perl.  A new
   release may come into existence in a near future, after a thorough
   review of the methods used, the set of errors checked for and some
   bug fixing (in particular, a Perl version of ddt-xfer is expected).
   In the mean time, the latest version is available from
   ftp://ns.dns.pt/pub/dns/ddt-2.0.1.tar.gz.

2.6. The Checker Project

   The problem of the huge amount of DNS traffic over the Internet is
   getting researchers close attention for quite some time, mainly
   because most of it is unnecessary.  Observations have shown that DNS
   consumes something like twenty times more bandwidth than it should
   [4].  Some causes for this undoubtedly catastrophic scenario lie on
   deficient resolver and name server implementations spread all over
   the world, from personal to super-computers, running all sorts of
   operating systems.

   While the panacea is yet to be found (claims are made that the latest
   official version of BIND is a great step forward [5]), work has been
   done in order to identify sources of anomalies, as a first approach
   in the search for a solution.  The Checker Project is one such
   effort, developed at the University of Southern California [6].  It
   consists of a set of C code patched into BIND's named, for monitoring
   server activity, building a database with the history of that
   operation (queries and responses).  It is then possible to generate
   reports from the database summarizing activity and identifying
   behavioral patterns from client requests, looking for anomalies.  The
   named code alteration is small and simple unless you want do have PEC
   checking enabled (see below).  You may find sources and documentation
   at ftp://catarina.usc.edu/pub/checker.

   Checker only does this kind of collection and reporting, it does not
   try to enforce any rules on the administrators of the defective sites
   by any means whatsoever.  Authors hope that the simple exhibition of
   the evidences is a reason strong enough for those administrators to
   have their problems fixed.

   An interesting feature is PEC (proactive error checking): the server
   pretends to be unresponsive for some queries by randomly choosing
   some name and start refusing replies for queries on that name during
   a pre-determined period.  Those queries are recorded, though, to try
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   to reason about the retry and timeout schemes used by name servers
   and resolvers.  It is expected that properly implemented clients will
   choose another name server to query, while defective ones will keep
   on trying with the same server.  This feature seems to be still under
   testing as it is not completely clear yet how to interpret the
   results.  A PEC-only error checker is available from USC that is much
   simpler than the full error checker.  It examines another name server
   client every 30 minutes to see if this client causes excessive load.

   Presently Checker has been running on a secondary for the US domain
   for more than a year with little trouble.  Authors feel confident it
   should run on any BSD platform (at least SunOS) without problems, and
   is planned to be included as part of the BIND name server.

   Checker is part of a research project lead by Peter Danzig from USC,
   aimed to implement probabilistic error checking mechanisms like PEC
   on distributed systems [7].  DNS is one such system and it was chosen
   as the platform for testing the validity of these techniques over the
   NSFnet.  It is hoped to achieve enough knowledge to provide means to
   improve performance and reliability of distributed systems.
   Anomalies like undetected server failures, query loops, bad
   retransmission backoff algorithms, misconfigurations and resubmission
   of requests after negative replies are some of the targets for these
   checkers to detect.

2.7. Others

   All the tools described above are the result of systematic work on
   the issue of DNS debugging, some of them included in research
   projects.  For the sake of completeness several other programs are
   mentioned here.  These, though just as serious, seem to have been
   developed in a somewhat ad-hoc fashion, without an implicit intention
   of being used outside the environments where they were born.  This
   impression is, of course, arguable, nevertheless there was no
   necessity of dedicating an entire section to any of them.  This
   doesn't mean they are not valuable contributions, in some cases they
   may be just what you are looking for, without having to install a
   complete package to do some testings on your domain.

   The reference taken was the contrib directory in the latest BIND
   distribution (where some of the above programs can also be found).
   There you will find tools for creating your DNS configuration files
   and NIS maps from /etc/hosts and vice-versa or generate PTR from A
   records (these things may be important as a means of avoiding common
   typing errors and inconsistencies between those tables), syntax
   checkers for zone files, programs for querying and monitoring name
   servers, all the small programs presented in [8], and more.  It is
   worth spending some time looking at them, maybe you'll find that
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   program you were planning to write yourself.  The latest public
   version of BIND can be found at
   ftp://gatekeeper.dec.com/pub/misc/vixie/4.9.2-940221.tar.gz.  As of
   this writing BIND-4.9.3 is in its final beta stages and a public
   release is expected soon, also at gatekeeper.dec.com.

   You may also want to consider using a version control system like
   SCCS or RCS to maintain your configuration files consistent through
   updates, or use tools like M4 macros to generate those files.  As
   stated above, it's important to avoid human-generated errors,
   creating problems that are difficult to track down, since they're
   often hidden behind some mistyped name.  Errors like this may end up
   in many queries for a non-existing name, just to mention the less
   serious kind.  See [9] for a description of the most common errors
   made while configuring domains.

3. Why look after DNS?

   Several pieces of software were presented to help people administer
   and debug their name services.  They exhibit many differences in
   their way of doing things, scope and requirements and it may be
   difficult just to choose one of them to work with.  For one thing,
   people's expectations from these tools vary according to their kind
   of involvement with DNS.  If you are responsible for a big domain,
   e.g., a top-level one or a big institution with many hosts and sub-
   domains, you probably want to see how well is the tree below your
   node organized, since the consequences of errors tend to propagate
   upwards, thus affecting your own domain and servers.  For that you
   need some program that recursively descends the domain tree and
   analyzes each domain per se and the interdependencies between them
   all.  You will have to consider how deep you want your analysis to
   be, the effects it will have on the network infrastructure, i.e.,
   will it generate traffic only inside a campus network, no matter how
   big it is, or will it be spread over, say, a whole country (of
   course, your kind of connectivity plays an important role here).

   You may simply want to perform some sanity checks on your own domain,
   without any further concerns.  Or you may want to participate in some
   kind of global effort to monitor name server traffic, either for
   research purposes or just to point out the "trouble-queries" that
   flow around.

   Whatever your interest may be, you can almost surely find a tool to
   suit it.  Eliminating problems like those described in this document
   is a major contribution for the efficiency of an important piece of
   the Internet mechanism.  Just to have an idea of this importance,
   think of all the applications that depend on it, not just to get
   addresses out of names.  Many systems rely on DNS to store, retrieve
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   and spread the information they need: Internet electronic mail was
   already mentioned (see [10] for details) and work is in progress to
   integrate X.400 operations with DNS [11]; others include "remote
   printing" services [12], distributed file systems and network routing
   purposes, among others.  These features may be accomplished by some
   standard, well-known resource records [13], or by new, experimental
   ones [14, 15].  Even if some of them won't succeed, one may well
   expect some more load on the DNS burden.

   The ubiquitous DNS thus deserves a great deal of attention, perhaps
   much more than it generally has.  One may say that it is a victim of
   its own success: if a user triggers an excessive amount of queries
   only to have one request satisfied, he won't worry about it (in fact,
   he won't notice it), won't complain to his system administrator, and
   things will just go on like this.  Of course, DNS was designed to
   resist and provide its services despite all these anomalies.  But by
   doing so it is frequently forgotten, as long as people can Telnet or
   ftp.  As DNS will be given new responsibilities, as pointed in the
   above paragraph, the problems described in this text will grow more
   serious and new ones may appear (notably security ones [16], with a
   lot of work being presently in progress addressing security in DNS),
   if nothing is done to purge them.
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Security Considerations

   Security issues are not discussed in this memo (although security is
   briefly mentioned at the end of section 3).
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